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ABSTRACT 



The National Postsecondary Education Cooperative (NPEC) , 
with the support of the U.S. Department of Education's National Center for 
Education Statistics, was formed to develop ways to improve the utility of 
information for policy making at all levels and in all sectors of the 
postsecondary education enterprise. This report provides a model for 
examining, from a policy perspective, postsecondary education data priorities 
in the student outcomes area. The report first identifies a range of policy 
issues currently facing American higher education. NPEC ' s "Student Outcomes 
from a Policy Perspective" Working Group developed the list. A taxonomy of 
student educational outcomes is then presented, and a procedure for linking 
the outcomes and policy taxonomies is presented as a first step in 
identifying those outcomes with the greatest potential for informing policy 
making in postsecondary education. A set of criteria for selecting specific 
outcomes within each outcomes domain is then introduced, as is a process for 
applying those criteria. The report closes with a series of recommendations 
for future steps toward the development of useful outcomes information. Four 
tables and one figure are included. Appendices contains elaborations of the 
proposed taxonomies of policy issues and student outcomes, an elaboration of 
the criteria for evaluating an outcome's policy relevance, and issues for 
future consideration. (Contains 34 references.) (LMI ) 
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STUDENT OUTCOMES INFORMATION FOR POLICY-MAKING 



In this ‘age of accountability, ’ administrators and others have been especially concerned about 
educational outcomes and their measurement. Postsecondary institutions . . . are also being called 
on to provide factual evidence that they and their programs are providing the benefits that were 
intended, and that these outcomes are being produced in a cost-effective manner (Lenning, 1977, p. 

ix). 

Lenning’ s description of some of the challenges facing postsecondary education in the U.S. may be 
nearly two decades old, but some things are slow to change. Currently, many members of Congress, 
governors, state legislators, top business executives, and others consider the information currently available 
on Americas postsecondary institutions to be inadequate for evaluating the quality of the education being 
delivered and for making decisions on policy or the expenditure of public funds (e.g., Romer, 1995). A 
number of forces have combined to increase the pressure on administrators and faculty members in public 
institutions to demonstrate the educational effectiveness of their institutions. Three of the most pressing 
include: 

• Increased competition for public funds from other public services (e.g., highways, 
prisons, social services) even as the resource base in most states erodes; 

• State appropriations that fail to keep pace with inflation; and 

• Legislative demands for accountability and increased productivity. 

While public institutions may feel these particular pressures more keenly than private institutions, 
other, equally-threatening sources of pressure make no distinction between public and private institutions in 
Americas postsecondary system. (“Postsecondary education,” as used here, refers to formal instruction 
beyond that provided in secondary school. It may be delivered in traditional colleges or universities, 
proprietary schools, tribal colleges, the military, corporate or industrial education programs, or other, similar 
settings. It does not include self-directed learning, such as occasional programs that may be delivered by 
libraries or museums, nor knowledge/skill development acquired through other informal, unstructured means.) 
Postsecondary educational institutions in the independent sector share with their public counterparts 
susceptibility to such pressures as: 

• The spiraling costs of attendance and operation; 
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Accrediting agency requirements to assess student learning outcomes and to 
incorporate the findings in operational and programmatic planning and decision- 
making; 



• Corporate and government leaders alarm about Americas economic competitiveness 
in a global market; 

• Writers both inside the academy (e.g., Anderson, 1992; Bloom, 1987; Huber, 1992; 

Smith, 1990) and out (e.g., Sykes, 1988) who have attacked college and universities 
reward structures and current directions; 

• News stories of financial mismanagement (and in some cases fraud) or educational 
malfeasance at both prestigious and obscure institutions; 

• Colleges and universities reluctance to engage in outcomes assessment and slowness 
in responding to external requests for outcomes information, which have fueled 
legislative and public frustration and impatience; 

• Graduates who cannot find postbaccalaureate employment despite a healthy 
economy; and 

• Employers increasing dissatisfaction with the skills their new employees bring to the 
workplace from their baccalaureate programs. 

Proprietary degree-granting institutions (perhaps because their programs are both shorter and more 
specifically targeted at employment than are those of not-for-profit institutions) appear to be less open to 
criticism relating to employer dissatisfaction, program completion, and employment placement rates than are 
institutions in other postsecondary sectors (e.g., Accrediting Commission of Career Schools and Colleges of 
Technology, undated). Their achievements notwithstanding, however, proprietary institutions confront many 
of the same educational “quality” issues facing their not-for-profit counterparts. 

The National Postsecondary Education Cooperative (NPEC), with the support of the U.S. Department 
of Educations National Center for Education Statistics, was formed to develop ways to improve the utility of 
information for policy-making at all levels and in all sectors of the postsecondary education enterprise. NPEC 
is a compact of data users and providers with representatives from a wide array of individual campuses, state 
higher education systems, and postsecondary education organizations, agencies, and accrediting bodies. One 
of NPECs first seven projects involves identification of student outcomes that could inform policy-making 
at the campus, state, and national levels and in all postsecondary sectors. A variety of institutional 
performance indicators are, of course, already available (e.g., admissions, registration, and financial aid 
systems at the institutional and state levels, and IPEDS at the national level). While such data are often useful 



for addressing some policy issues, the information they yield tends to be restricted to students precollege 
characteristics and various “production counts” (e.g., students enrolled, credit-hours produced, degrees 
awarded). Current student data systems yield little information about an institutions educational effectiveness 
—the outcomes it produces. Such questions as “What and how much do students learn from any given 
postsecondary education experience?” go largely unaddressed by current student data systems. To answer 
such questions, one must look beyond this operational perspective to examine current and projected 
institutional, state, and national student outcome data needs and deficiencies in a policy context. The 
purpose of this report is to provide a model for examining, from a policy perspective, postsecondary 
education data priorities in the student outcomes area, as well as (at a later point in time) in other areas 
such as instructional costs, postsecondary access, workforce development, and new delivery systems. 

The report first identifies a range of policy issues currently facing American higher education. NPECs 
Student Outcomes from a Policy Perspective” Working Group developed the list. A taxonomy of student 
educational outcomes is then presented, and a procedure for linking the outcomes and policy taxonomies is 
presented as a first step in identifying those outcomes with the greatest potential for informing policy-making 
in postsecondary education. A set of criteria for selecting specific outcomes within each outcomes domain 
is then introduced, as is a process for applying those criteria. The report closes with a series of 
recommendations for future steps as work on the development of useful outcomes information for policy- 
making proceeds. 

A Taxonomy of Policy Issues 

The introductory portion of this report described a number of the pressures pushing postsecondary 
education toward student outcomes assessment and a common set of outcome variables and measures. That 
discussion, of course, does not exhaust the list of pressing issues confronting the postsecondary system. To 
bring some conceptual order to the many and varied policy issues that are current in postsecondary education 
or might be anticipated for the future, Robert A. Wallhaus drafted a taxonomy of policy issues as a starting 
point for discussions of the Student Outcomes from a Policy Perspective Working Group. The original draft 
was subsequently revised by Wallhaus and Patrick T. Terenzini, and revised still further by the Student 
Outcomes from a Policy Perspective Working Group. 
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Table 1 displays that taxonomy of policy issues. The taxonomy is intended to be relevant to all 
postsecondary levels and sectors, whether public or private, proprietary or not-for-profit. It contains fifteen 
categories, and, as might be expected, some of the issue areas are more likely than others to be informed by 
student outcomes information. Appendix A contains a more detailed discussion of the meaning of each policy 
domain label and provides additional examples to clarify the information provided in Table 1. 

Defining a Student Outcome 

Before consensus can emerge on what outcomes to include in an outcomes-information-for-policy 
system, the meaning of the term “outcome” must be defined. The Working Group uses “outcome” to refer 
to those education-related consequences of students postsecondary educational experience. The outcomes 
with which this report is concerned are those believed to be attributable to the postsecondary experience rather 
than to normal maturation, societal changes, or other influences or forces beyond postsecondary educations 
sphere of influence. 

Outcomes may be short- or long-term, occurring during or after a students postsecondary experience. 
They may be direct (i.e., an immediate consequence of some postsecondary educational experience) or indirect 
(i.e., due to postsecondary educations effect on some intervening variable that is more directly related to an 
outcome). Outcomes may also be intended (e.g., cognitive development, employability) or unintended (e.g., 
enhanced athletic ability, marriage, dependence on chemical substances). A substantial number of outcomes, 
of course, might be identified as potentially associated with participation in a formal postsecondary education 
process. For that reason, the outcomes taxonomy proposed is restricted to those outcomes that are typically 
intended outcomes of some postsecondary education experience and within the power of postsecondary 
institutions to shape to educational advantage through programmatic or policy interventions. 

A Proposed Taxonomy of Intended Student Outcomes 

Most of what we currently know about what happens during, and as a consequence of, students formal 
educational experiences has been developed from studies of traditional undergraduate college and university 
students (see, for example, Feldman & Newcomb, 1969; Pascarella & Terenzini, 1991). One might reasonably 
argue, however, that most current taxonomies of postsecondary student outcomes are sufficiently generic to 
be applicable to a wide variety of diverse learners and teaching/leaming contexts. What varies, this line of 



3 

ERIC 



4 



TABLE 1. A TAXONOMY OF POLICY ISSUES 



INPUT ISSUES 

Access: Diversity (broadly defined), minority participation (including students, faculty, and staff), affirmative 
action, immigration, program availability, geographic accessibility, transfer and articulation, access to 
the benefits of postsecondary education 

Affordability: Costs to student/family, sources and adequacy of financial aid, family income and ability to pay, 
students education-related indebtedness, loan default rates 

Financial Support: Trends in sources and amounts of revenues, cost of instruction, privatization, total cost, level 
and composition of private/state/federal funding for research, indirect-cost recovery, performance-based 
funding 

Student Preparation for Postsecondary Education: Need for remediation, admission standards, 

secondary-postsecondary linkages, teacher preparation 

PROCESS ISSUES 

Accountability: Overall institutional quality, assessment policies, performance indicators (e.g., persistence, 
time-to-degree, degree completion, job placement rates), governance, regulatory requirements and 
reform, consumer disclosure and choice 

Campus Climate: Racial/ethnic and gender relations, sexual harassment, crime, speech codes 

Facilities and Equipment: Condition of space, capital investment, technological up-grades, deferred 

maintenance 

Faculty: Workloads (volume and nature), reward systems, tenure, compensation, relations of faculty research 
to state needs, distribution of full-/part-time faculty, supply and demand, diversity, faculty development 

Productivity: Program duplication, elimination of administrative bloat, cost savings, efficiency (e.g., retention, 
time-to-degree, degree completion), cost-effectiveness, return on investment, incentives for improvement 

Technology: Access to information; learner access and productivity; investment costs; impact on current policies, 
organizational structures, financing and accounting mechanisms, instructional support services, and 
personnel; curriculum and credentialling control; impact on course content, instructional methods, role 
of faculty, employment skills, and assessment 

END-PURPOSE ISSUES 

Continuing Education and Life-Long Learning: A number of issues related to distance education, but also 
to more generic matters, including content, place of instruction, when learning occurs, how it occurs, 
who does the teaching, who sponsors it, and who credentials it. 

Public Service/Outreach: Agricultural extension, land-grant colleges, rural health care, community and 
national service 

Quality/Educational Effectiveness: Student achievement and other outcomes (what and how much is 
learned), learning productivity 

Quality of Graduate and Professional Education: Heterogeneous issues, many similar to those relating to 
undergraduates (e.g., access, quality, productivity, workforce preparation and retraining) 

Workforce Preparation and Retraining: Occupational supply/demand, employer expectations, educational 
and training opportunities, geographic access, postsecondary- workforce articulation, job/employment 

rates, economic development 
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argument maintains, is the degree to which learners change on any given outcome, the sources of influence 
on those changes, and the value or worth individuals or institutions attach to a particular change as a legitimate 
educational outcome to be promoted. Moreover, while the area, degree, and sources of change might vary 
substantially with student characteristics (e.g., gender, race/ethnicity, age, ability, learning style), the learning 
setting (e.g., method of instruction, learner activities, content and/or skills to be learned), or the postsecondary 
education sector, the possible dimensions along which a learner may change might usefully be thought of as 
relatively common across postsecondary learners, settings, and sectors. 

Appendix B summarizes three popular outcomes taxonomies the Working Group considered. Each 
has its strengths and weaknesses, depending upon the use for which it was intended. Astins (1976, 1993) 
model was created to guide the selection of variables for a large, national research program. Ewells (1984) 
classification was developed to help guide institutions in their development of student outcomes assessment 
programs, and the NCHEMS structure (Lenning, 1977) was constructed to provide a common vocabulary for 
the “full range” of postsecondary education outcomes data collection, exchange, and analysis in support of 
evaluation and planning, primarily at the institutional level. 

\ 

Following a review of the structure and primary characteristics of each of these typologies, the Student 
Outcomes Information from a Policy Perspective Working Group concluded that none of these typologies was 
satisfactory for purposes of identifying student outcome variables likely to be useful for policy-making at all 
levels and in all postsecondary sectors. Astins four-fold model was judged to be too general, while the 
NCHEMS structure was considered overly and, for present purposes, unnecessarily complex. Ewells 
classification has much to recommend it, although some members considered it not sufficiently detailed as 
a structure with which to begin a process of consensus building on specific, policy-relevant outcomes that 
might become part of an outcomes-information-for-policy system. 

The proposed taxonomy is intended to be generic, describing educational outcomes which one or more 
types of postsecondary institution seek to achieve. The importance any given institution attaches to a specific 
outcome will, of course, vary with the postsecondary sector, institutional mission, and clientele. Thus, it is 
important in this discussion to differentiate an educational outcome or goal from the routes institutions in 
various sectors of the postsecondary system may take to achieve their goals and desired outcomes. The 
taxonomy is intended to be applicable and useful to all postsecondary institutions, but not all parts of it will 
be equally important or relevant to all institutions, even within the same sector. 
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Table 2 offers a twelve-domain outcomes taxonomy. The structure was initially developed by Robert 
A. Wallhaus for the NPEC Student Outcomes Information from a Policy Perspective Working Group and was 
subsequently revised by Wallhaus, Patrick Terenzini, and the Working Group. The overlap with Ewells 
(1984) classification (see Appendix B) is apparent, although certain outcomes (e.g., educational success, 
success in transitions, quality of life) subsumed under one of Ewells major categories are given greater 
prominence in the proposed taxonomy. In addition, the proposed model (unlike Ewells) also includes a 
category for “Economic Benefits.” It should be noted that the Working Group does not consider the first three 
outcome domains listed in Table 2 (“Communication and Computational Skills” through “Content Learning”) 
to be unrelated to “Occupational Preparation” and “Workplace Skills” outcomes. Those three domains are 
highly associated with job performance. More detailed descriptions and examples of the kinds of outcomes 
comprising each domain are given in Appendix C. 

Given the proposed taxonomy of student educational outcomes, the challenge remains to develop 
consensus on this structure and — more importantly for any effort to develop an outcomes-information-for- 
policy system -- to find a means for linking relevant outcomes information with policy issues that are 
important currently or are likely to be in the future. The next several sections describe a set of related 
procedures for making such linkages. 

Linking Student Outcomes with Policy Issues 

Having developed what were considered adequate taxonomies of major policy issues and student 
outcomes, the Student Outcomes Information from a Policy Perspective Working Group set about the task of 
identifying those outcomes they considered to be most relevant to one or more of the policy issues they had 
identified. As the process of identifying policy-relevant outcomes progressed, however, it became readily 
apparent that Working Group members had different conceptions of what an “outcome” is. Astins (1993) 
input-environment-outcomes (or I-E-0 model) was introduced as a means of clarifying and differentiating 
outcomes from the experiences or processes that produce those outcomes. 

Figure 1 presents a schematic representation of Astins model, which contains three sets of constructs 
or variables involved in studying and measuring the influences on students postsecondary experiences on 
student growth or change. In particular, this model is a useful tool for identifying and estimating the effects 
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TABLE 2. A TAXONOMY OF STUDENT OUTCOMES 



Educational/Training Achievement 

Academic 

Communication and Computational Skills: Reading, writing, and oral communication; quantitative/ 
computational skills; information acquisition skills (technological and otherwise) 

Higher-Order Cognitive and Intellectual Development: Critical thinking, problem solving, analytical and 
evaluative skills, formal and postformal reasoning, conceptual complexity, creativity, moral reasoning (as 
a process) 

Content Learning: General (breadth) and specific (depth) of knowledge 

Occupational 

Occupational Preparation: Knowledge and skills specific to an occupation; occupational choice; occupational 
status; job placement; licensure; job satisfaction; performance; productivity; promotability; occupational 
mobility; employer satisfaction, occupational aspirations 

Workplace Skills: Motivation to perform in the workplace, dependability, adaptability, persistence, initiative, 
leadership skills, ability to work independently and in groups 

Developmental 



Psychosocial Development: Autonomy, tolerance for diversity, intellectual orientation, interpersonal skills and 
maturity, motivation (generic), identity development, self-concept and self-esteem, personal adjustment 

Attitudes, Values, and Beliefs: Occupational, educational, cultural (arts), social, political, religious, interpersonal 
(e.g., diversity), standards of conduct, orientation to life-long learning 

Civic Development: Group affiliations/memberships, citizenship, community involvement, voting participation 

Attainment of Student Goals 

Educational Success: Retention/persistence, educational aspirations, educational attainment, course/program/ 
degree completion, time-to-degree, satisfaction 

Success in Transitions: Education-to-work, education-to-education, work-to-education 

Economic Impacts: Income, return on investment, standard of living, geographic mobility, educationally-related 
financial indebtedness 

j2uallty_of Life: Sense of well-being, health, consumer behaviors, savings and investment behaviors, leisure activity 
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Figure 1 




Astin’s l-E-0 Model 



of those postsecondary experiences over which administrators or policy makers might exercise some 
programmatic or policy control to enhance educational effectiveness. 

The inputs component of Astins model includes a wide variety of personal, family, and educational 
background characteristics that students bring with them to their postsecondary experience. These traits 
include such things as academic and intellectual abilities, precollege achievements (academic and otherwise), 
goals and motivation levels, degree and career aspirations, and a range of demographic, personal, and family 
characteristics, such as gender, race/ethnicity, socioeconomic status, age, marital status, and citizenship. 
“Inputs” also refers to a variety of other obligations students may have, including those to family and/or work. 
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The environment of Astins I-E-O model includes all those things that might influence what and how 
much students learn or change. At least four general, institutional-level sources of influence on student 
outcomes are identifiable: Curricular influences (e.g., courses taken, major field), formal instructional 
experiences (e.g., type and quality of instruction, interaction with faculty in class), out-of-class experiences 
(e.g., peer relations, place of residence, informal contact with faculty, work and/or family obligations), and 
institutional characteristics (e.g., type and control, curricular mission, size, selectivity, environment or 
culture). Institutional characteristics would also include the human, financial, and physical resources an 
institution has available to it. These resources shape the educational environment and students responses to 
it in both obvious and subtle ways. Student effort is yet another relevant variable. While one might consider 
it an input, the case can be made that the effort students exert during their college experience (and in response 
to it) is a different characteristic from the initial level of motivation students bring to college. In any event, 
student effort has been well established as an important influence on the degree of student change or 
development (Pascarella & Terenzini, 1991). 

These environmental variables — whether at a given institution or across institutions -- comprise most 
of the programmatic and policy levers available to campus administrators and policy makers to shape students 
postsecondary experiences and, thereby, the outcomes of postsecondary education. 

Outcomes in Astins I-E-O model correspond directly to the student outcomes discussed earlier and 
listed in Table 2. They are, quite simply, the “products” of students postsecondary experiences, the effects 
of attending a postsecondary institution. 

Astins model is a parsimonious -- but far from simple — representation of the dynamics and processes 
of postsecondary impact on students. The model specifies that the background traits and experiences students 
bring with them to a postsecondary setting (i.e., the “inputs”) have both direct and indirect effects on 
outcomes. The effect of background characteristics on outcomes is direct (indicated by path C in Figure 1) 
in that the best predictor of an educational outcome (say, level of cognitive development) is the students level 
on that variable at the time the postsecondary experience was begun. Background characteristics, however, 
also shape outcomes indirectly through their influence on the nature of a students encounter with the 
“environment” of the postsecondary setting, which, in turn, affects outcomes. This influence is represented 
by path A in Figure 1. The postsecondary environment and students experiences in it, in turn, directly affect 
outcomes (path B). 
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While it was never intended for this purpose, Astins I-E-O model can be used to advantage in linking 
policy issues and student outcomes. Specifically, the model provides a logic for identifying those policy issues 
that can be illuminated by student outcomes information. The close correspondence between the categories 
of policy issues given earlier (input, process, and end-product issues; see Table 1) and Astins I-E-O model is 
readily apparent and provides a basis for linking policy issues and student outcomes. The “fit,” however, is 
sometimes imperfect. 



The final step in the process of identifying policy-relevant student outcomes information requires the 
conjunction of the two taxonomies as a means of bringing some focus to the discussion and selection of the 
most salient, policy-related student outcome variables. Just as it was apparent that not all policy issues were 
equally susceptible to illumination by student outcomes information, however, the Working Groups 
examination of the potential linkages between policy issues and student outcomes led to recognition that not 
all outcome domains are equally relevant to end-product policy issues. Within each policy category, the 
Working Group evaluated and determined the level of relevance of each outcome domain. Table 3 presents 
a policy-by-outcome matrix that summarizes the results of the policy-outcomes linking process. “Primary” 
and “Secondary” designations reflect the Working Groups perceptions of the degree of centrality of each 
outcome to the policy debate. Different postsecondary institutions with different missions will, of course, 
attach greater importance to some outcomes than to others. What may be “primary” for one institution may 
well be “secondary” (if not irrelevant) for another. Recognizing these imperfections in the assignments 
reported below, the Working Group nonetheless believes some such differentiation is important if any degree 

of parsimony is to be achieved. Without such differentiation, one would have to consider all outcomes to be 
equally central to the policy debate. 

The policy issues-by-outcome-by-relevance framework accomplishes several objectives. First, it adds 
parsimony. Within the End-Purpose Issues category (the policy areas to which outcomes information is 

arguably the most central), it reduces the domains of the original student outcomes taxonomy from thirteen 
to the “primary” seven. 



Second, despite the potential points of debate, the framework offers a starting point for a process of 
reaching consensus on which outcome domains should be given the most attention. Such a process can lead 

to further discussions on more precise identification and specifications of variables within each domain (see 
below). 
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TABLE 3. LINKING OUTCOMES AND POLICY ISSUES 





Relevance of Outcomes Information 


Policy Issues 


Primary 


Secondary 


End-Purpose Issues 




1 .. • ' . ' 7 ■ 


Quality and Educational 
Effectiveness 


Communication and 
Computational Skills 
Cognitive and Intellectual 
Development 
Content Learning 
Occupational Preparation 
Workplace Skills 
Educational Success 
Success in Transitions 


Attitudes, Values, and Beliefs 
Economic Benefits 
Social Development 
Psychosocial Development 
Civic Development 
Quality of Life 


Workforce Preparation and 
Retraining 


Primary Outcomes as Above 
Workplace Skills 


Secondary Outcomes as Above 
(minus Workplace Skills) 


Quality of Graduate Education 
(selected) 


Primary Outcomes as Above 
(as they relate to graduate ed.) 


Secondary Outcomes as Above 
(as they relate to graduate ed.) 


Continuing Education and Life- 
Long Learning (selected) 


Primary Outcomes as Above 


Secondary Outcomes as Above 


Process Issues 






Accountability (e.g., assessment 
policies, regulatory 
requirements and reform) 


Primary Outcomes as Above (re. 
adequacy of current policies 
and practices) 


No Others Relevant 


Campus Climate (selected) 


Attitudes, Values, and Beliefs 
Psychosocial Development 
Civic Development 


No Others Relevant 


Facilities and Equipment 


None Relevant 


None Relevant 


Faculty Issues 


None Relevant 


None Relevant 


Productivity (selected) 


Primary Outcomes as Above 


Secondary Outcomes as Above 


Public Service/Outreach 


None Relevant 


None Relevant 


Technology (learning-related uses) 


Primary Outcomes from 

End-Product Categories (above) 


Secondary Outcomes from 
End-Product Categories (above) 


Input Issues * - 


Access 


None Relevant 


None Relevant 


Affordability 


Student Indebtedness 
Economic Impacts 
Educational Success (persistence, 
degree completion, time-to- 
degree) 


No Others Relevant 


Financial Support 


None Relevant 


None Relevant 


Student Preparation 


None Relevant 


None Relevant 
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Third, the framework identifies a number of postsecondary education policy issues that are clearly 
important and urgent, but which may be more precisely thought of as sources of influence on student outcomes 
(i.e., part of the postsecondary educational “environments”), rather than as outcomes themselves. 

Criteria for Evaluating an Outcome's Policy-Relevance 

In order to specify more precisely the appropriate outcomes information for policy making, it will be 
necessary to reach some consensus on what variables within each outcomes domain should be included. 
Making those selections requires a set of criteria or principles for choosing among alternative possibilities. 
In the present case, criteria may be of at least two separate, but interrelated, kinds: conceptual and 
methodological. 



Conceptual criteria involve philosophical and/or political considerations. They can be thought of as 
a set of issues relating to the question “Why should this outcome be included in the data set under 
development?” Methodological criteria involve technical issues of measurement, availability, and data 
collection design. The criteria listed in Table 4 constitute the set of considerations the Working Group 
believes should be taken into account in evaluating the merits of an outcome variable for policy-making. The 
criteria are categorized into three “screens,” the purpose of which is described in the next section. (Appendix 
D provides a more detailed description of each criterion.) 

Applying the Criteria 

The final challenge is to apply the proposed criteria to each of the policy and outcome domains and, 
in the process, to refine outcome variable specifications within each domain “in such a way that the resulting 
measures remain credible across a diverse postsecondary community and with such key external audiences 
as employers and policy makers” (Ewell, 1996, p. 23). 



Applying the criteria will require reaching some consensus on which outcomes within each domain 
should be evaluated for possible inclusion in the data set. The goal of this process would be to identify 
outcomes that are both more specific and more narrowly defined than those currently listed in the outcomes 
taxonomy. In the case of the “higher-order cognitive and intellectual development” domain, for example, a 
number of possible skills are listed (e.g., critical thinking, problem-solving, analytical and evaluative skills). 
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TABLE 4. CRITERIA FOR EVALUATING POLICY-RELATED OUTCOME VARIABLES 



First-Screen Criteria (Conceptual Considerations) 

• Relevance: How closely is the outcome related to an important policy issue? 

• Utility: What is the potential of information on this outcome for guiding action to achieve policy 
objectives? 

• Applicability: To what extent will information on this outcome meet the needs of multiple 
stakeholder/user groups? 

Second-Screen Criteria (Conceptual Considerations) 

• Interpretability: Will information on this outcome “communicate” with users? 

• Credibility: How believable is information on this outcome likely to be for policy makers? 

• Fairness: Will the information bias or mislead in ways that may disadvantage one group of 
stakeholders or users vis-a-vis another? 

Third-Screen Criteria (Methodological Considerations) 

• Scope: Are the needed data “census-type” (i.e., drawn from all institutions and students) or 

“knowledge-base” (i.e., drawn from another data collection design, such as a sample survey)? 

• Availability: Do data on the outcome already exist? Are they accessible? Does a “technology” exist 
for collecting data on the outcome? 

• Measurability: How is the outcome to be operationally defined and measured, and how 

methodologically sound are those measures likely to be? How much error can be tolerated in the 
measure being evaluated, given the policy question(s) being addressed? 

• Cost: How expensive will it be to obtain information on the outcome? 



Which of those are to be included in the data set: All of them? Some of them? Which ones? To apply the 
criteria, a finer-grained level of specification is needed in all domains. Although greater specification risks 
some loss of consensus in trying to put too fine a point on these selections, such refinement would appear to 
be a necessary and logical step. 

The Working Group believes that the inherent conceptual and policy value of an outcome should take 
precedence in the selection process over technical considerations. Methodological matters are nontrivial ones, 
of course, but outcomes found to be strong on a number of the conceptual criteria ought not to be rejected for 
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being methodologically problematic. Rather, ways should be found to develop appropriate measures or 
procedures for obtaining useful information for policy-making. 

To apply the “three-screen” process, selection panels could be used both to specify outcomes more 
precisely and to evaluate their merits for policy-making. These panels would consist of individuals 
representing diverse data user and supplier groups (from both inside and outside postsecondary education) and 
would make the initial selection of specific outcomes within a specific domain (i.e., one panel for 
communication skills, another for computational skills, another for higher-order cognitive and intellectual 
development, and so on). It may be prudent to include on each panel at least one individual who is familiar 
with current state and national data systems, as well as at least one person with strong research design and 
analysis skills and experience in studying the impacts of college on students. 

The “first-” and “second-screen” criteria deal with possible answers to the question of why any given 
outcome variable might be important for policy-making. An outcome judged wanting in terms of the first 
screen (i.e., relevance, utility, or applicability) is probably not worth considering further. Outcomes passing 
that screen, however, should then be examined on the criteria comprising the second screen: Interpretability, 
credibility, and fairness. 

Assuming the first- and second-screen evaluations indicate an outcome has promise, an outcomes 
“third-screen” merits should be evaluated. These methodological criteria involve process considerations 
related to data collection, measurement, and the human, technical, and financial resources required to assemble 
information on an outcome. Whereas the first- and second-screen criteria addressed questions of why 
information on an outcome variable might be important to policy-making, these third-screen address questions 
of how — and how easily — information on an outcome might be gathered, how sound those data are likely 
to be, and how costly gathering the data might be. 

One should not expect any outcome variable to “score” highly on all criteria at each screening level. 
The screens provide a hierarchical framework for evaluating whether a variable is “worth keeping around” 
for further consideration. In the end, variable selection will require judgments of a variables overall “success” 
across all criteria. 
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At each screening level, panelists might undertake their evaluation of the merits of each outcome 
independently of one another. Panelists would have a worksheet for each outcome, listing all criteria and 
providing space for panelists to record written comments, as well as a summative rating (e.g., low, medium, 
or high) for each criterion. Panelists might also be asked to give an overall summative rating for each outcome 
at each screening level. At each level, panelists would then share their evaluations of each outcome with other 
panelists. The strengths and weaknesses of each outcome at that screen-level would be discussed, panelists 
explaining the reasons for their evaluations and benefiting from the thinking and judgment of other panel 
members, until consensus has been reached. The same process would then be repeated at the next screening 
level until consensus on an outcomes overall merits has been reached and a decision made on whether to 
include it in the data set to be developed. 

Project Accomplishments and Implications 

The Working Groups activities have, we believe, led to several accomplishments with implications 
for future NPEC projects. First, the Taxonomy of Policy Issues (see Table 1) has considerable promise for 
structuring discussions about data and information needs to inform policy-making in areas other than student 
outcomes. For example, discussions of policy-related data needs in such areas as academic and non-academic 
personnel, equipment acquisition, facilities, or financial support can be given focus and direction using this 
taxonomy. To a certain degree, the policy taxonomy can function as a touchstone forjudging whether data 
systems (current or future) speak to policy issues. 

Second, the Taxonomy of Student Outcomes (see Table 2) is the first of its kind specifically to address 
policy concerns. The taxonomy provides a categorization of student educational outcomes that is not only 
more comprehensive and specific than its predecessors, but also general enough to promote discussion and 
avoid definitional disputes. 

Third, the process that led to the cross-referencing of policy issues and relevant student outcomes 
information in this project (see Table 3) can be readily applied in future efforts to identify and link relevant 
variable sets (e.g., in academic and non-academic personnel) to policy issues. 

Fourth, the set of criteria for evaluating the policy-relevance of student outcomes variables can also 
be readily applied to variable sets in other areas, such as personnel, facilities, finances, or accounting. While 
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the word “outcome” appears throughout that list of criteria, the underlying construct in each case is 
generalizable to variables of almost every kind. 

Finally, the proposed process for applying those criteria is also generalizable to policy-related data 
discussions in most areas of postsecondary education. One of its major attractions is that variable selection 
and operationalization is that it involves “representatives” of major stakeholder, data supplier, and data user 
groups in a collaborative, consensus-building process. 

Recommendations 

Based on the foregoing, the Working Group offers the following recommendations: 

1 . The feasibility of the criteria and procedures for identifying specific and appropriate 
outcome variables proposed in this report should be validated. 

We believe the outcomes-by-policies model (see Table 4) has demonstrated its utility by 
identifying plausible linkages between student outcomes variables and relevant policy issues. 
We also believe the criteria and process for evaluating outcomes variables provide a workable 
basis for selecting and operationalizing outcome measures to inform relevant policy debates 
and decision-making, but that belief has yet to be established. Are the criteria comprehensive 
and conceptually sound? Is the proposed process feasible and effective for delineating more 
specific outcomes variables? Are the criteria appropriate in number and sufficiently and 
meaningfully defined? 

2. Funding should be provided to support a pilot test of Recommendation 1 in a limited 
number of outcomes areas. 

We believe it is prudent to test the questions posed in Recommendation 1 on a limited basis 
before attempting to apply the criteria on a more ambitious scale. We believe, moreover, that 
appropriate funding will be needed to undertake such a pilot. The pilot test should involve 
two selection panels (as outlined in this report) to evaluate the conceptual adequacy of the 
model and the applicability of the criteria. The charge to the selection panels in this pilot test 
would be to select a limited number of outcomes variables and recommend how they should 
be operationalized. We recommend pilot testing in two areas (one panel per outcome area) 
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of the policy-outcomes matrix: 1 ) Occupational Preparation, and 2) Cognitive and Intellectual 
Development. We recommend the first area because it is one about which a good deal is 
known, and because instruments and data collection methods are already available in certain 
subcategories of that domain. In contrast, much less is known about the assessment of 
cognitive development on a large scale, particularly as this domain relates to policy making. 
We believe pilot testing in two domains that are sharply different with respect to available 
information and data collection methodologies will afford a realistic test of the model and 
criteria under different and important conditions. We also believe that external stakeholders 
(e.g., business leaders and other non-academic and non-NCES individuals) should be 
involved in the pilot project. 

3. The stated purposes of the Student Outcomes from a Policy Perspective project have 
been completed, with one exception. Achievement of that particular goal (because it 
overlaps the charge of another NPEC project group) should be undertaken jointly by 
the Student Outcomes from a Policy Perspective Working Group and the Student 
Outcomes from a Data Perspective Working Group within the context of the pilot. 

Initially, the Student Outcomes from a Policy Perspective project had four goals: “1) define 
policy issues at the national, state, and institutional levels which are concerned with student 
outcomes; 2) identify data which, if available, would lend insights into the various policy 
issues; 3) develop a conceptual framework for relating student outcomes data to specific 
policy issues; and 4) assess approaches to developing data priorities from a policy perspective 
and determine deficiencies in student outcomes data.” The Working Group believes it has 
achieved Goals 1 , 3, and 4 and begun the process of achieving Goal 2. Goal 2, however, 
requires efforts extending beyond this projects scope, and, to some extent, it overlaps the 
charge of NPECs Student Outcomes from a Data Perspective project. The Policy Working 
Group believes, however, that the pilot test proposed in Recommendation 2 can be an integral 
part in the achievement of Goal 2 and sees that goals full achievement as representing and 
requiring the conjunction of the work of the two groups. In our view, the pilot test constitutes 
a vehicle for uniting the goals and energies of the policy perspective and data perspective 
groups, which leads us directly to our next recommendation. 
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4. The pilot test selection panels should include representatives from both the “policy 
perspective” and “data perspective” Working Groups, as well as representatives from 
appropriate information user- and provider-groups. 

The proposed panels will, we believe, require representation from both the “policy 
perspective” and “data perspective” groups in order to maintain some continuity with the 
work these two groups have completed thus far. At the same time, however, it is well to 
remember that the implementation of a broadly usable outcomes information system is an 
inherently political process. Thus, the work of the selection panels is likely to benefit from 
(if not require) augmentation with representatives of data users and providers in the policy 
arena, as well as one or more individuals with significant expertise in measurement and 
research methodologies. Selection panels so constituted, we believe, are more likely than a 
group with a more restricted membership to produce recommendations that 1) meet the 
information needs of policy makers, 2) are methodologically sound and practicable in the 
eyes of information providers, and 3) will, consequently, meet with broader acceptance 
among policy information users and providers. 

j. In constituting selection panels, care should be taken to ensure that the purview of each 
panel is conceptually reasonable. 

Certain of the outcome domains in the taxonomy developed by this project group are rather 
broad conceptually. For example, the “Communication and Computational Skills” domain 
comprises several, quite different subcategories of abilities (e.g., writing, reading, speaking, 
and computation). It seems unlikely that a selection panel could include representatives of 
data user- and provider-groups from important postsecondary institutional and policy sectors, 
as well as the necessary measurement and research experts, without becoming unmanageably 
large and probably counterproductive. When an outcomes domain is broadly inclusive, we 
urge that discrete panels be constituted so as to recognize important - but policy-relevant - 
distinctions between and among outcomes. 
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6. 



Efforts should be undertaken to explore the relevance and utility of the policy issues 
taxonomy and the process outlined in this report for linking policy issues with other 
relevant kinds of postsecondary education information (e.g., faculty or personnel 
information, budget and/or facilities information, and environmental influences on 
student learning). 

The policy taxonomy appears to be generalizable across postsecondary sectors (e.g., types 
of institutions, missions, clienteles). Similarly, the process of linking policy issues with 
relevant kinds of information (in this case, outcomes information) also appears to be a 
generalizable procedure that might be usefully applied to other types of postsecondary 
education information. We believe it will be useful in the future, however, to think of the 
process of linking issues with information, and then applying the evaluative criteria 
developed in this report, as two distinct processes. Our experience suggests that the policy 
taxonomy, an information taxonomy, and the merger of the two are relevant, useful, and 
productive undertakings. As suggested in Recommendation 1, however, we are less certain 
of the extent to which the evaluation criteria developed in this project and their application 
to the intersections of policy and information are reasonable and workable in other 
informational sectors. 

7. Policy issues, not methodological issues, should drive considerations in all NPEC efforts 
to develop taxonomies, information models, and data collection plans. 

As indicated in this reports discussion of the need for two levels of conceptual criteria before 
consideration is given to methodological issues, the Working Group believes strongly that 
policy issues, not methodological considerations, should be the determining factor in the 
development of postsecondary education information systems. Information judged in the 
abstract to be relevant, useful, and applicable to important policy matters should not be 
ignored or dismissed because of its unavailability, problematic measurement, or (within 
reason) cost. Instead, ways should be sought to overcome obstacles to providing information 
that is needed and relevant to one or more important policy issues. 
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8 . 



The Working Groups discussions surfaced a variety of relevant issues for future 
consideration. While beyond the Groups charge, they are important issues that should 
not be overlooked. They include: (Appendix E contains a more detailed description of each 
issue) 

a) The appropriate unit of analysis; 

b) The importance and policy relevance of inputs and environmental influences; 

c) Distinctions between “value-added” and “status-description” data needs; 

d) Building on what we already know; 

e) Variations in information needs according to functional responsibility and 
organizational level; 

f) Possible development of an outcomes performance “index”; and 

g) Responsibility for data collection. 
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APPENDIX A 



ELABORATION OF THE PROPOSED TAXONOMY OF POLICY ISSUES 



Input Issues 

Access, like student preparation (to which it is closely linked), has long been an issue for 
postsecondary education. Access issues have arisen and evolved as federal and state policy decisions over the 
past forty years have changed the missions of higher education and the demographic characteristics of the 
students it serves. “Access” refers to both a condition and a process. As a condition, the term refers to the 
extent to which postsecondary education is available to those who wish to take advantage of it. The root issue 
is one of equity and the equality of opportunities to participate. Access is a particular concern as it relates 
groups of citizens who have traditionally been excluded or whose access has been constrained. Such groups 
include not only those based on race/ethnicity or gender, but also on age, employment or socioeconomic 
status, physical or learning disability, geographic location, and so on. 

“Access” as a condition is a policy matter in at least two senses. First, the term may apply to the 
equality of opportunities to enroll in some form of postsecondary experience or to attend some institution in 
a state higher education system. Virtually every state now provides equality of access in one fashion or 
another. “Access,” however, can also refer to the equality of students opportunities to benefit from 
participation in postsecondary education (Pascarella & Terenzini, 1991). 

In terms of public policy, “access” can also refer to the processes by which institutions and 
governments seek to remove barriers from postsecondary education based on any of the conditions listed 
above. Processes affecting equal access include not only conventional marketing, recruiting, and admissions 
standards, policies, and practices, but also legal requirements (e.g., affirmative action and other equal 
opportunity legislation) intended to extend access to all who can benefit from postsecondary education. 
Policies governing interinstitutional transfer and articulation agreements constitute another subset of the 
processes influencing access to postsecondary education. 

Affordability policy issues are, of course, closely related to those of access, although the former have 
become more urgent as the costs of postsecondary education have risen and sources of support have tended 
to dry up. “Affordability,” in this policy taxonomy, is seen as a set of issues relevant most immediately to 
individual students and their families, but also indirectly to all state and federal taxpayers and their elected 
officials. Affordability topics include costs to students and/or families, the sources of financial aid and their 
adequacy, levels of schooling-related student and family financial indebtedness, and student loan default rates. 
These issues are likely to become even more salient than currently to the extent that federal efforts are 
successful in shifting more of the responsibility for supporting students to the states. Affordability, thus, is 
seen as a set of issues distinct from — but by no means unrelated to — Financial Support issues (see below). 

Financial Support policy issues, in contrast to those of affordability, deal directly with institutional 
matters (e.g., endowments, philanthropy, grants, contracts, tuition) and governmental funding rather than with 
personal or individual financial matters. The issues are ones of both source and level of support for 
postsecondary education. This category includes such questions as how much aid state and/or federal 
governments should spend on postsecondary education as opposed to other public needs, what the sources of 
that aid should be, the distribution of financial support across different categories of institutions (e.g., flagship 
campuses vs. other institutions in a state system) and students, and whether a particular service (e.g., student 



health care) should be provided by educational institutions or through private sources. This category also 
includes a host of other questions relating to the costs and funding of instruction and other postsecondary 
functions and services. 

Student Preparation for Postsecondary Education issues are philosophical and political, as well 
as educational. States are wrestling with questions concerning the quality of their K-12 systems. These 
debates breed additional questions about the quality of teacher preparation programs and, in turn, questions 
about which level of their K- 1 6 systems should be responsible for preparing students to do college-level work. 
(Should colleges and universities be offering remedial education to high school graduates?) Efforts to answer 
these questions encounter still other, important, tightly-interlaced, public policy issues. What should be the 
standards of admission to postsecondary education? What should be the relationships between K-12 and 
postsecondary education? Who should set the standards? 



Process Issues 



Accountability includes a range of policy issues, not all of which are related to student outcomes. 
As a general policy construct, the term refers to the responsibility (if not legal obligation) of campus and 
system administrators, as well as governmental officials, to provide superiors (ultimately the public) reports 
of their stewardship of public funds. Such officials have always had a professional responsibility to account 
for their use of public dollars, but since the mid-1970s, deteriorating state and national economic conditions 
have generated demands for greater accountability Institutional and state education officials are under pressure 
to provide evidence that their institutions are producing outcomes consistent with their mission and worth the 
cost. Such demands raise a host of difficult philosophical, political, and methodological questions: Who 
should decide (and how) what the goals of a public institution or system should be? Who should decide (and 
how) what the definition and standards for “worth” should be? How are costs to be measured and then 
associated with outcomes (see Bowen, 1974)? Postsecondary institutions are audited annually in many areas 
of operation, of course, but calls for “accountability” in the present context are less concerned with tracing the 
precise expenditure of funds than with the justification of those expenditures. Accountability issues involve 
documenting operational and educational cost-effectiveness (see Productivity below), as well as matters of 
governance, reporting and related regulatory requirements and reform, and assessment policies (a prominent 
subset of regulatory requirements). Few campus, state, or federal officials and observers expect the pressures 
for accountability to diminish anytime soon. 

Campus Climate issues and conditions mirror those in the society at large and have attracted greater 
attention in the wake of increasing incidents of harassment or outright violence based on gender, 
race/ethnicity, sexual orientation, religion, or other differences in the personal characteristics or beliefs of those 
involved. Perceptions of, and concern over, growing campus crime rates have also helped promote the belief 
that Americas postsecondary institutions are increasingly more dangerous, intolerant places than they were 
at almost any time in their history. In the policy arena, efforts to promote greater tolerance through the 
promulgation of speech codes collide with First Amendment rights. Individual rights to safety and privacy 
clash with Constitutional protections against unreasonable search and seizure. Campuses are struggling to 
build a sense of community as law makers search for ways to a “kinder, gentler” world both on and off 
campus. 



Facilities and Equipment issues always arise as a consideration when budgets become tight. While 
capital construction on most campuses has been curtailed, problems of deferred maintenance and equipment 
replacement costs (both favorite budget-balancing tools) continue to require attention. Indeed, given the 
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current rates of technological development and enhancement, equipment maintenance and replacement costs 
(as well as associated facilities modification costs) are likely to become more urgent problem areas. 

Faculty Issues have come in for closer scrutiny because faculty salaries comprise such a large portion 
of any postsecondary institutions operating budget and because faculty are central players in the 
teaching-learning process. Both the nature of faculty work (e.g., the distribution of effort between teaching 
and research) and its volume (both absolute and relative effort) have received, and are likely to continue to 
receive, close examination in a context characterized by wide divergence of opinion in philosophies of 
education, institutional missions, and state and national needs. Closely related to the issues of the nature and 
volume of faculty workload are questions about the appropriateness of current faculty reward structures, the 
policies and practices involved in the promotion and tenure processes, and, indeed, the need for tenure itself. 
Debates continue over the use of part- vs. full-time faculty, the gender and racial/ethnic composition of most 
faculties, and concern is rising over whether doctoral production in some disciplines is appropriate to the 
demand for graduates. Other issues relate to whether faculty development efforts constitute an effective means 
of reducing instructional costs and increasing instructional efficiency and effectiveness. 

Productivity may refer to the total volume of production, but as the term is used in postsecondary 
settings, it also implies “efficiency,” the relationship between what is produced and the cost of producing it. 
In postsecondary education (as in any business), productivity can be enhanced by any of three achievements: 
higher production at a steady cost, stable production at reduced cost, or increased production at lower cost. 
Enhancing productivity might involve elimination of program duplication either on the same campus or across 
campuses in a region or state. It might be related to growth in productivity through higher persistence and 
degree-completion rates. It can also take various forms of cost savings, as in privatizing or out-sourcing 
certain operations (e.g., residence halls, food service, bookstores, health services). Gains in productivity may 
also be realized through the use of incentives to enhance programs or operations. 

Levin (1971) has suggested three categories of “efficiency.” Operational efficiency (what most of us 
understand “efficiency” to mean) is the relation between inputs and outputs. Given the inputs, were maximum 
outputs achieved? A second for of efficiency, allocative efficiency , however, concerns the extent to which 
greater outcomes might have been achieved by varying the mix of inputs, the allocation of resources. Finally, 
says Levin, one might also speak of preference efficiency , which refers to the extent to which a particular mix 
of outputs was the most valuable combination for society: Were the right goals pursued? 

Advances in Technology have put a host of policy, educational, and administrative issues on the table 
In some cases, the issues overlap with those in other categories. Educational policy questions in this domain 
include how best to provide student and faculty access to the new information technologies, the instructional 
effectiveness of those new technologies (and how to assess it), how course credit will be awarded (e.g., when 
students take courses at a distance from another institution), how an institutions control of its curriculum can 
be maintained without stifling the creativity and flexibility of the new technologies, and the implications for 
changes in postsecondary educations credentialling function. Advances in information technologies have also 
raised significant questions about course content, instructional methods, and the appropriate roles of faculty 
members and students. Administrative policy questions will impact current organizational structures- 
budgeting, financing, and accounting policies for allocating costs and distributing cost-savings, and 
instructional development and production support for faculty, students, and staff who use the new 
technologies. 





End-Purpose Issues 



Continuing Education and Life-Long Learning issues concern, to a large extent (but not 
exclusively), matters relating to standards and accreditation for learning programs that are technology-driven 
(primarily distance education efforts). In many states, institutions are reluctant to accept courses that are 
available “at a distance” without some assurance of high quality standards. Should guidelines for setting those 
standards differ from those for more traditional, nontechnologically-supported courses? Can existing 
accrediting associations reasonably be expected to address recognition/certification issues of programs offered 
at a distance dispersed among a number of institutions (e.g., the Western Governors University)? 

Continuing education and life-long learning issues, however, extend beyond those that are technology- 
related. Spille (cited in Wallhaus, 1996) has suggested seven major changes, already underway in the U.S., 
that will require adaptations in current instructional content, structures, and delivery methods. These changes 
include those in 1) what is taught and learned; 2) where learning occurs; 3) when learning occurs; 4) how 
learning occurs; 5) who teaches; 6) who sponsors the learning, and 7) who credentials the learning. 

Public Service/Outreach policy issues relate to the nature and extent of the role of postsecondary 
institutions (e.g., land grant colleges, their agricultural extension units, community colleges) in delivering (or 
assisting in the delivery of) services to local, state, and regional communities. The policy issues include 
provision of cultural enrichment to local communities, turf issues relating to the delivery of services, equity 
in the geographical distribution of service providers vis-a-vis areas in need of such services, and declines in 
financial support for cooperative extension efforts even as expectations grow for the delivery of such services 
in urban and inner-city settings. Whether, when, where, and how national and community service should be 
part of the undergraduate curriculum constitutes yet another area of concern. 

Quality and Educational Effectiveness contains the set of policy issues involving questions of what 
and how much students are learning and the extent to which that learning is benefiting both individuals and 
the public. The particular areas of learning and their related benefits are listed in the outcomes taxonomy 
presented earlier. “Learning productivity” (the what-is-leamed-for-how-much ratio), is included in this 
category rather than as a special subset of the Productivity domain (see below) because of its definitional links 
to outcomes. 

Quality of Graduate and Professional Education contains a heterogeneous set of policy issues 
closely related to many of those under discussion for undergraduate education. These issues include matters 
of access, quality, productivity (particularly return on investment), financial support (both the adequacy of 
current funding and the distribution of funding between teaching and research), and workforce preparation 
and retraining. Other graduate or professional policy issues concern the payoff for the research being done, 
market demand for graduate training in some disciplines, and the payoff of graduate education to the 
individual student. 

Workforce Preparation and Retraining. Preparation of the workforce (in some sectors, at least) 
has been a central mission of higher educational institutions since the founding of Harvard College, which was 
intended (among other things) to train physicians, clergy, and government leaders for the new colony. The 
importance of workforce preparation to the public good has been periodically reaffirmed since that time 
through a variety of legislative and Congressional actions. Calls for postsecondary education to be responsive 
to state and national needs have grown in recent years. They are unlikely to recede as the nation struggles to 
regain its international economic competitiveness and to reduce the national debt. 
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Workforce policy issues affecting postsecondary education in the U.S. are many, varied, and rapidly 
changing. Driven by a changing economy and emerging technologies, workforce supply and demand, 
employer expectations of (and satisfaction with) new and continuing employees, educational training programs 
and opportunities, equal access to employment, and economic development. 



APPENDIX B 



OTHER TAXONOMIES OF STUDENT OUTCOMES 



A variety of student learning outcome taxonomies have been advanced. They range from those 
specific to a particular outcome domain, such as Blooms (1956) taxonomy of cognitive outcomes to 
Chickerings (1969) seven “vectors” of students psychosocial development (see also Chickering & Reisser, 
1993). Any extended explication of available taxonomies is beyond the scope and purpose of this report, but 
it will be useful to examine some in order to appreciate the range of their complexity and inclusiveness. Doing 
so, it is hoped, will illuminate the possibilities from which one might choose in developing a state or national 
student learning outcomes data set. 



Astins Four-Fold Typology 

Astin (1976, 1993) recognized that “There is no way to capture the impact of college adequately in 
one or two simple measures, such as credits and degrees earned or job placement” (1993, pp. 8-9). To guide 
variable selection for the Cooperative Institutional Research Program (CIRP) data collections, he developed 
the typology shown in the accompanying table. The framework is based on the type of outcome and the type 
ofdata to be gathered. Types of outcomes were categorized into “two broad domains: cognitive (sometimes 
called intellective) and noncognitive (sometimes called affective)” (1993, p. 9). Types ofdata were similarly 
dichotomized into “psychological data, relating to the internal states or traits of the individual; and behavioral 
data, relating to directly observable activities” (1993, p. 9). 

Astins typology is perhaps the most parsimonious available. While the two “type” categories 
(cognitive and noncognitive) have been used for decades in the behavioral sciences, the reduction of types of 
outcomes to two domains offers little guidance for the selection of more specific outcome variables for use 
in the policy arena. Similarly, the “types of data” dimension has more utility for variable selection and data 
collection than for identification of specific outcomes. That dimension is, however, a useful reminder that 
student outcomes may be behavioral as well as psychological. 



Ewells Classification of Outcome Dimensions 

Ewell (1984) has offered a taxonomy more detailed than Astins. It contains four general categories 
of outcomes: knowledge, skills, attitudes and values, and relationships with society and particular 

constituencies. Each major domain has two or more subcategories which both help explain the content of a 
particular outcome domain and offer more specific examples of the classes of variables contained within each 
domain. The text accompanying the outline of Ewells dimensions (pp. 31-44) provides additional examples 
of the content of each category. 

Unlike Astin (1976, 1993), Ewell (1984) gives no attention in his typology to issues of the data to 
be collected or to the time dimension. His categories, however, provide substantially more detailed explication 
of the domains of student outcomes. For this reason, Ewells framework has greater potential than Astins for 
identifying and categorizing student outcome variables that might be useful in the policy arena. 



The NCHEMS Outcomes Structure 



The taxonomy developed by Lenning and his colleagues (Lenning, 1977; Lenning, Lee, Micek, & 
Service, 1977) for the National Center for Higher Education Management Systems (NCHEMS) is by far the 
most complex and inclusive. The NCHEMS structure was intended to “serve as a framework for organizing 
information in an effective way for purpose of classification, analysis, and decision-making” Lenning, 1977, 
p. ix). It was designed not as a “student outcomes” taxonomy, but as a “framework that will accommodate 
information about the full range of postsecondary education outcomes” (Lenning, Lee, Micek, & Service, 
1977, p. 1). Student outcomes comprise only a portion of the NCHEMS taxonomy. 

The NCHEMS structure (figure) rests on three dimensions. The first is an audience dimension (a 
“first level of detail”) that defines those individuals or groups who are touched in some way by postsecondary 
educations outcomes. The dimension has five second-level categories: individuals/group clients; 

interest-based communities; geographic -based communities; aggregates of people (i.e., groups based on some 
distinguishing personal characteristic, such as ability, age, gender, race/ethnicity, socioeconomic status), and 
other audiences (e.g., the environment or populations of animals). The structure further defines 28 
subcategories within the five major groups of the audience. 

The second NCHEMS structural dimension, type of outcome , names and defines the nature of 
postsecondary outcomes. As with the audience dimensions, the type of outcome dimension is subdivided into 
five major outcome domains: economic, human characteristic (other than knowledge and understanding); 
knowledge, technology, and art form; resource and service provision, and other maintenance and change 
outcomes. These five major categories are themselves further subdivided into 24 second-level and 83 
third-level subcategories. The similarities between Ewells (1984) five-group classification and the NCHEMS 
structures first three second-level categories (excepting the technology and art form portions of the third 
category) are apparent. 

A time dimension comprises the third portion of the NCHEMS structure and refers to the expected 
or actual occurrence of an outcome. This dimension was not subdivided because it was believed no set of 
categories would apply across audience groups. Lenning and his colleagues included because timing 
considerations are so important for planning and evaluation. 
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APPENDIX C 



ELABORATION OF THE PROPOSED 
TAXONOMY OF STUDENT OUTCOMES 



Communication and Computational Skills refers to the foundational skills students will need to 
function in the modem world and which will facilitate subsequent, self-directed learning. This domain 
includes skills in reading, oral and written expression, numeric calculations, and information acquisition, 
including the use of libraries, information technologies, and listening). 

The Higher-Order Cognitive and Intellectual Development domain comprises a set of general 
intellectual and cognitive competencies and skills. These skills constitute more complex mental functions than 
those in the Communication and Computational Skills domain. These skills generally involve the effective 
processing and utilization of information and include such activities as critical thinking, problem-solving, the 
analysis and evaluation evidence or information, creative thinking, and both formal (e.g., Piagetian) and 
postformal reasoning (the analysis of problems for which there is no verifiably-correct answer, sometimes 
called ill-structured or wicked problems, such as crime, poverty, abortion). This category also includes 
moral reasoning, which refers to the processes by which judgments of right and wrong are made, not the 
contents of students beliefs about what is moral. 

Content Learning refers to the subject matter knowledge students develop during their postsecondary 
educational experience. This domain has two general subcategories: The first is the “breadth” component of 
most undergraduate students curricular experiences. It consists of a general and broadly-based body of 
knowledge on which there is some consensus on what students should know. In traditional colleges and 
universities, students are exposed to this knowledge base through “general education” or “core” curricular 
experiences. What this “common” knowledge should be, of course, has been the subject of heated discussions 
in this country for nearly two centuries. The second, or “depth,” category comprises the knowledge (and 
skills) needed to achieve a higher level of mastery and competence in a specific subject area. It is the content 
and competencies of the “academic major field.” The relations of this category of content and skills to 
occupational or vocational preparation may be highly variable, depending on a students occupational goals. 

The Occupational Preparation domain comprises the knowledge and skills students will need to 
obtain employment and function effectively in a specific vocation, occupation, or profession. The nature and 
effectiveness of the preparation for work students receive while in college have implications for career choice, 
occupational attainment, career progression and success, occupational mobility, job satisfaction, occupational 
status, and the transfer of that status advantages from one generation to another. 

Workplace Skills refers to a more generalized set of personal characteristics and interpersonal 
competencies that people need to function effectively in most work settings. This category includes such 
personal traits as dependability and adaptability, as well as such interpersonal skills as the ability to work in 
groups and leadership skills. 

Educational Success includes a set of outcomes having to do with educational aspirations and 
students progression toward their education goals. It includes such variables as degree aspirations, persistence, 
time-to-degree, educational attainment (number of years of schooling completed or highest degree completed)! 
and degree completion (completion of one or more certificate or degree programs). 
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Success in Transitions has to do with the extent to which one or more postsecondary education 
experiences equip students to move successfully (and with minimum difficulty) between educational and/or 
occupational settings, or to function in an educational and an occupational setting simultaneously. A transition 
may involve movement from education to employment, from employment to education, or from one 
educational setting to another (e.g., from an associate degree program to a baccalaureate program, or from an 
undergraduate program to graduate or professional school). This outcome has become increasingly important 
as postsecondary learning and retraining activities become more integrated. This outcome has been ignored 
in previous taxonomies. 

Economic Benefits refers to the direct financial and related benefits that accrue to a students 
postsecondary educational experience. Such benefits include income, the return on investment, standard of 
living, level of educationally-related financial indebtedness, and the ability to move geographically in order 
to obtain more remunerative employment. 

Psychosocial Development comprises a range of personal dimensions that can be subdivided into 
two categories. The first involves “internal” changes in individuals relations with themselves (the “psycho-” 
portion of “psychosocial”). The postsecondary experience has been shown to produce changes in an 
individuals “sense of self’ in such areas as personal identity, self-concept, self-esteem, personal adjustment 
or maturity, and general psychological health (Pascarella & Terenzini, 1991). Psychosocial development may 
also involve changes in individuals orientations to their “external” world (the “-social” portion of the 
adjective). The postsecondary experience might lead to changes in individuals interpersonal skills, tolerance 
for people different from themselves and for ideas different from their own, or their intellectual curiosity 
(included in this category because it seems to be clearly neither an academic “content” nor a higher-order 
cognitive “skill”). In general, the facets of development covered by this domain tend to be relatively “central” 
to an individuals being, as well as enduring and slow to change. 

While Attitudes and Values might be considered part of students psychosocial makeup, they are 
treated separately in this taxonomy because of the expanse of this domain and because one might reasonably 
argue that what one refers to as “attitudes” or “values” are more specific in nature and less resistant to change 
than the personal dimensions included under the “psychosocial” rubric. This domain comprises such 
outcomes as an appreciation of the visual and performing arts, occupational and educational values (e.g., the 
importance attached to intrinsic vs. extrinsic rewards), sociopolitical attitudes and values, religious orientation 
and values, interpersonal orientations (e.g., toward peoples of other races and cultures, gender, or sexual 
orientations), and standards of conduct. 

Civic Development, one might reasonably argue, should be considered one dimension of students 
attitudes and values. As used here, however, the term refers to a set of specific, other-person-oriented 
behaviors, rather than to the more general dispositions or inclinations that are characteristics of ones attitudes 
and values. The outcomes listed above in those two domains may or may not be associated with actual 
behavior. “Civic development” refers to how one, in fact, behaves toward others, with strong overtones of 
community membership, participation, and responsibility. 

The Quality of Life domain includes a variety of postsecondary education outcomes that are 
noneconomic in nature but nonetheless real. Such outcomes include a generalized “life satisfaction,” health, 
marriage, child nurturance behaviors (e.g., in matters of health and education), consumer behaviors, savings 
and investment behaviors, and leisure time activities. Caution is in order, however, when attempting to refine 
and define these outcomes more precisely. While certain of these outcomes (e.g., life satisfaction and health) 
probably cut across social classes and cultures in terms of their desirability, the value attached to other 
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outcomes (e.g., smaller families, long-term perspective in investment strategies) may vary by social class or 
socioeconomic status. 





APPENDIX D 



ELABORATION OF THE CRITERIA FOR EVALUATING 
AN OUTCOME’S POLICY-RELEVANCE 



First-Screen Criteria 

Relevance deals with how closely an outcome is related to an important policy issue. “Relevance” 
reflects the strength of the conceptual linkage between an outcome variable and one or more policy issues. 
It addresses the appropriateness of the outcome for informing policy discussions. Consideration should also 
be given, however, to the “relevance” of an outcome for an institutions mission. Is the outcome consistent 
with what the institution says it is trying to accomplish? 

Utditv concerns the extent to which information on the outcome is likely to be useful for informing 
policy-making or for evaluating a particular policy that already exists. “Utility” refers to the potential of the 
information the outcome can provide for guiding action to achieve a policy objective. Utility may, of course, 
vary across user groups. The greater the number of user groups for whom an outcome is useful, the greater 
its utility. 

Applicability refers to the extent to which the information to be collected meets the needs of multiple 
stakeholder/user groups. The various groups to be served include not only policy makers at various state or 
national levels, but also across sectors of postsecondary education and across levels or categories within any 
given sector. 



Second-Screen Criteria 

Interpretabilitv refers to the clarity or conceptual complexity of the outcome. How readily understood 
is information on this outcome likely to be to various user groups? How well will information on the outcome 
“communicate” with multiple and diverse audiences (Ewell, 1994)? For example, information on specific 
economic outcomes such as employment, salary, or job status are likely to be readily understood by various 
audiences. In contrast, information on outcomes whose definitions are complex, slippery, or in dispute (e.g., 
cognitive development, creativity, or principled moral reasoning) may present interpretation problems for 
users. 



Credibility reflects the degree of trust different stakeholder groups might be expected to place in 
information on an outcome. The term refers to the “face validity” of an outcome - the extent to which an 
outcome appears to measure what it is supposed to measure, although it may not actually do so. Credibility 
is, of course, rooted in the values and perceptions of the data users, and it will no doubt vary across 
stakeholder groups. 

Fairness refers to what Ewell (1994) has called the “balance of perspective,” the extent to which the 
outcome “reflect(s) the perspectives and concerns of multiple constituents.” The important consideration is 
that use of an outcome not present biased or misleading information that serves to give one or more 
stakeholder or user groups an advantage over other groups. 
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Third-Screen Criteria 



Scope has to do with whether “census” or “knowledge-base” type data are needed, whether data are 
needed from all institutions on all students (as in IPEDS) or from only a representative sample of institutions 
and/or students (as with High School and Beyond). Not all policy debates require census-type data, and 
collection of census data on some outcomes (e.g., cognitive gains) may well be a practical impossibility. 
Resolution of this consideration will have significant implications for data collection designs and costs. 

Availability refers to the ease with which information on an outcome can be obtained. The concept 
has at least three dimensions. The first issue is the very existence of data on a desirable outcome (Does it exist 
at all?). The second dimension is the accessibility of the data. For example, students scores on a particular 
test may exist but access to them may be denied or restricted in some fashion (e.g.. Graduate Record 
Examination scores). The third issue, feasibility, concerns the extent to which a “technology” already exists 
for measurement and data collection. Do instruments for data collection already exist? Have appropriate and 
workable data collection designs been developed? Is the hardware and software available for processing the 
data collected? 

Measurability has multiple dimensions, most of which relate to the canons of measurement in the 
social sciences. The three most important issues, of course, are the operational definition to be adopted, the 
reliability alternative measures (the accuracy and consistency of measurement), and the validity of those 
measures (the extent to which a measure assesses what it is believed to assess). One additional facet of the 
validity of an outcome measure to be used in the policy arena is what Ewell (1994) has called its 
“vulnerability,” the extent to which the measure might be manipulated to suggest change that has not occurred 
in what the measure is believed to reflect. Measurability, however, also has a value-judgment dimension: 
What is the appropriate level of precision? Increased precision usually comes with a higher price tag, and as 
some point, decisions will have to be made about whether the increase in precision is worth the increased cost. 
This judgment must be made in light of the potential consequences of being wrong (i.e., of tolerating too much 
error in a measure): How much precision is required, given the policy issue, and (conversely) how much error 
can be tolerated? 

Cost refers to the likely financial expenditures of collecting information on the variable being 
evaluated. Costs will originate with decisions concerning data collection design, size of the target sample or 
population, length of the data collection cycle (e.g., one-shot, periodic, annual), instrumentation (development 
and/or purchase), and staffing. 
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APPENDIX E 



ISSUES FOR FUTURE CONSIDERATION 



The charge to the Working Group was to define important student outcomes-related policy issues 
facing postsecondary education, to develop a framework for relating outcomes data to specific policy issues, 
and to assess approaches to developing data priorities from a policy perspective. As might be expected, the 
Group members and report reviewers raised a number of related issues. We have tried to address those that 
were clearly related to the Working Groups charge. Other issues, while not so directly relevant to the Groups 
business, are nonetheless important. They are described briefly here in the hopes that the issues will not be 
lost in subsequent discussions and activities as this project moves forward. 

a) Appropriate Unit of Analysis. The Working Group has focused its attention on outcome variables 
that might be measured with the student or institution as the unit of analysis. Certain policy issues, 
however, may require more aggregated, or “societal-level” outcomes information (e.g., the 
characteristics and distribution of individuals with particular workforce skills, general occupational 
or educational attainment levels for specific geographic or racial/ethnic groups). While societal-level 
statistics might be constructed from individual-level data, it will be important during the variable 
selection and subsequent data collection design processes to consider the specific data needs of the 
policy issues to be addressed and the appropriate unit of analysis for developing those data. 

b) Importance of Inputs and Environmental Influences. The Working Groups charge was to focus 
on student outcomes, but as Astins Input-Environment-Outcome(I-E-O) Model makes clear (see 
Figure 1), outcomes are a function of the characteristics students bring with them to postsecondary 
education (the inputs) and of the characteristics of the institutions they attend and the kinds of 
experiences they have while there (the environments). As Ewell (report review) has pointed out, 
“most policy questions are in fact situated inside the causal arrows of the I-E-O model rather than 
lying within its domain-oriented boxes.” Policy making seeks to shape behavior, and if educational 
outcomes are to be purposefully and positively shaped, information about the sources of influence on 
those outcomes (i.e., on inputs and environments) will be needed. Outcomes data collection 
development efforts should be coordinated with any future NPEC efforts relating to gathering 
information about relevant input and environment variables. These two sets of variables (and 
environments more so than inputs) constitute the major foci for policy intervention. 

c) “Value-Added” and Status-Description Issues. “Value-added” issues concern the extent to which 
students change as a consequence of their postsecondary education experience. Information on the 
nature and degree of such longitudinal change may be needed for certain policy decisions (e.g., 
whether to continue to invest in particular educational interventions or programs). Other policy 
questions, however, may require information that is more descriptive and cross-sectional (e.g., the 
distribution of particular skills in the workforce across segments of the population or in the population 
as a whole). Attention should be given to the distinctions between “growth” and “adequacy/capacity” 
measures and their intended use for informing policy. 

d) Building on What We Know. The process of selecting and building consensus on outcome variables 
need not start from scratch. The National Center for Education Statistics (1992, 1994, 1996) has 
sponsored two conferences which attempted to define the outcome domains for national data- 
collection purposes. Several reports by the National Center on Postsecondary Teaching, Learning, 
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and Assessment (Jones, 1996, 1997; Jones, et al., 1994; Jones, Dougherty, Fantaske, & Hoffman, 
1997) describe Delphi-based studies of the definitional consensus among faculty, administrators, and 
employers on the National Educational Goals of critical thinking, communication, and problem 
solving. Travers (1996) describes potential performance indicators in mathematics, and Adelman 
(1989) reports on similar efforts in computer science, mechanical engineering, biology, physics, and 
chemistry. 

Similarly, much is known about the sources of influence on student outcomes (e.g., Pascarella & 
Terenzini, 1991). A useful next step might be to prepare brief, policy maker-oriented literature 
reviews of what we know about colleges influences on student change and development. 

e) Variations in Information Needs According to Functional Responsibility and Level. Information 
needs will vary with both the functional responsibility of the individual (e.g., administrator, trustee, 
legislator, staff member) and the organizational level at which the person serves (e.g., institution, state 
system, region, state or federal policy maker). The underlying question is “Who needs what and at 
what level of detail?” The Working Group was unable to address this question in the time available 
to it, but it is a matter that should be considered at some future stage as this project evolves. 

f) Development of an Outcomes Performance “Index”? Morgan Lewis (an outside reviewer) raised 
the question of developing a performance index for institutional, state, or national outcomes 
performance that would be analogous to the Dow-Jones Index, the Consumer Price Index, or similar 
indices (e.g., employment rates) for evaluating the health of the nation’s economy. Such an index 
would consist of a limited number of “face-valid outcomes, especially for those with legislative 
responsibility” (review, p. 2). The Working Group did not consider such a possibility. It is included 
here (consistent with the title of this section) as one more “issue for future consideration.” 

g) Responsibility for Data Collection. The Working Group was not charged with addressing a series 
of questions raised by one reviewer. They are fundamentally important questions, however, and are 
included here to “keep them on the table.” At some future point, NPEC will have to come to grips 
with them: “Who will produce this information, for whom, to be used for what purpose? ... [If 
campuses are to be the producers,] Who will ask them [to] produce more, and by what authority, to 
what end, and with what trade-offs? ... In a national data-collection effort, whose responsibility 
will it be to tum that data into information?” (Margaret Miller). 
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